Security

None.

Review metrics

Merge-risk options

Maintainer options:

1. Route local state checks through Hermes (recommended)
   Keep the adapter and its context available during scoring, then add coverage for a local state assertion that proves the adapter verifier—not the gateway shim—decides the result.
2. Limit the PR to file and execution tasks
   Explicitly restrict the generic local path to file and execution verification until adapter-aware state scoring is implemented, accepting a narrower Hermes capability surface.

@clawsweeper automerge

Special instructions:
Keep the local adapter context available through scoring, route state assertions to its verifier, and add a regression test for an adapter-backed state assertion.

Technical review

Best possible solution:

Score local adapter runs while the live adapter context is available, passing state assertions through the adapter’s verifier and preserving OpenClaw’s existing gateway-based scoring path.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection gives a high-confidence path: select Hermes ai_agent mode for a task with a supported state assertion, then observe that scoring receives _LocalVerificationClient rather than the live Hermes adapter verifier.

Is this the best way to solve the issue?

No—the generic adapter lifecycle is the right direction, but scoring after teardown with a gateway-only client is not the narrowest correct integration; state verification must use the live adapter context.

Full review comments:

• [P1] Route state assertions through the live adapter — clawbench/harness.py:640-644
  The local path calls teardown before scoring and passes _LocalVerificationClient to score_task_run, so Hermes state assertions never reach HermesAdapter.verify_state_query. In ai_agent mode Hermes advertises memory, session, and cron support; those tasks can pass capability gating but be scored against an unavailable gateway. Score before teardown or thread the adapter/context into verification, with a regression test for a state assertion.
  Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: The patch enables a useful benchmark adapter path, but its remaining defect is bounded to correctness of local-adapter state scoring.
• merge-risk: 🚨 session-state: Merging as written can score Hermes memory, session, and cron assertions against an unavailable gateway instead of the agent state that ran the task.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.

Evidence

Acceptance criteria:

• [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q.
• [P1] python -m pytest -q.
• [P1] Run one redacted local Hermes ai_agent stateful task and capture terminal output showing the state assertion result.

What I checked:

• Current main does not implement Hermes execution: The current harness explicitly rejects every registered adapter except OpenClaw and tells users that Hermes is not wired into end-to-end scoring; the adjacent test asserts that adapter="hermes" raises that error. (clawbench/harness.py:124, 884dd1bb5511)
• PR bypasses the adapter state verifier: The branch leaves the adapter context and calls teardown before scoring, then supplies _LocalVerificationClient to score_task_run; that shim raises for gateway state calls instead of invoking adapter.verify_state_query. (clawbench/harness.py:640, 67ff264005ab)
• Hermes advertises state capabilities and implements verification: In ai_agent mode, Hermes declares memory and cron capabilities and implements verify_state_query for memory, session, and cron checks, so routing those checks through a gateway shim produces the wrong evaluation boundary. (clawbench/adapters/hermes.py:249, 884dd1bb5511)
• Adapter contract establishes the intended lifecycle: The adapter base contract says state queries are resolved with adapter.verify_state_query(query, ctx) before adapter teardown; the PR has not implemented that contract in the generic scorer path. (clawbench/adapters/base.py:20, 884dd1bb5511)
• Relevant feature history: The adapter canonicalization layer was introduced by 56531fbf432b97db1fc14b0254efb9d858eeb679, and recent current-main harness work includes workspace isolation in 9fe22f9233f723e5554e7bef01b75daf54cfcba5. (clawbench/harness.py:124, 56531fbf432b)

Likely related people:

• scoootscooob: Introduced the canonical adapter layer that defines the lifecycle and capability boundary this PR extends. (role: adapter-layer introducer; confidence: high; commits: 56531fbf432b, f09a9f4bf74b; files: clawbench/adapters/base.py, clawbench/adapters/hermes.py, clawbench/harness.py)
• Robin Narsingh Ranabhat: Most recent locally available current-main history touching the harness focused on per-run workspace behavior, which this local-adapter path also uses. (role: recent harness contributor; confidence: medium; commits: 9fe22f9233f7; files: clawbench/harness.py)
• Vincent Koc: Recent harness history includes scoring and runtime hardening changes relevant to preserving correct benchmark results. (role: recent scoring and runtime contributor; confidence: medium; commits: fb486a1ed307, ed9adf8d8451; files: clawbench/harness.py, clawbench/scorer.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Route state assertions through adapter.verify_state_query before teardown.
• Add a focused regression test for a supported local memory, session, or cron assertion.
• Rebase the repaired path against current main and post redacted live output for that stateful task.

Rating scale

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History